Skip to content

Publish the libraries a few at a time - #351

Merged
sjoelund merged 1 commit into
masterfrom
faster-publishing
Sep 4, 2026
Merged

Publish the libraries a few at a time#351
sjoelund merged 1 commit into
masterfrom
faster-publishing

Conversation

@sjoelund

@sjoelund sjoelund commented Sep 4, 2026

Copy link
Copy Markdown
Member

Once the tests are done, the run publishes one library at a time, and what that costs is not the copying but waiting for the far side: each library asks rsync four times over ssh, and the one that matters lists a remote directory of tens of thousands of files to honour --delete-excluded.

Three things were paid for per library and are now paid for once:

What Before After
git log of this repository once per lib once
git log of the reference files once per lib once per dir
rsync connections to make the dirs 3 1

The directories are made in one connection by telling rsync -R where the relative part of the source path starts, so emptydir/./<lib>/files creates the branch, the library and its files directory at once instead of one rsync per level.

The libraries are then published four at a time. Not more: each one is an ssh connection of its own, and sshd allows ten unauthenticated connections by default.

The loop body became a function, which also stops it shadowing the stats dictionary of the run with a list of its own.

Assisted-by: Claude Opus 5

Once the tests are done, the run publishes one library at a time, and
what that costs is not the copying but waiting for the far side: each
library asks rsync four times over ssh, and the one that matters lists
a remote directory of tens of thousands of files to honour
`--delete-excluded`.

Three things were paid for per library and are now paid for once:

| What                               | Before       | After        |
| ---------------------------------- | ------------ | ------------ |
| git log of this repository         | once per lib | once         |
| git log of the reference files     | once per lib | once per dir |
| rsync connections to make the dirs | 3            | 1            |

The directories are made in one connection by telling rsync -R where
the relative part of the source path starts, so `emptydir/./<lib>/files`
creates the branch, the library and its files directory at once instead
of one rsync per level.

The libraries are then published four at a time. Not more: each one is
an ssh connection of its own, and sshd allows ten unauthenticated
connections by default.

The loop body became a function, which also stops it shadowing the
`stats` dictionary of the run with a list of its own.

Assisted-by: Claude Opus 5
@sjoelund
sjoelund enabled auto-merge (squash) September 4, 2026 18:47
@sjoelund
sjoelund disabled auto-merge September 4, 2026 19:05
@sjoelund
sjoelund merged commit 283f231 into master Sep 4, 2026
6 of 7 checks passed
@sjoelund
sjoelund deleted the faster-publishing branch September 4, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant